Interesting question. As we know the astronauts who visited the moon took pieces away in the form of rock and soil samples. They now sit in several museums. I guess it really depends on the size of the piece you remove. If a chunk with a mass of thousands of kilotons where removed it would affect the moon's gravity as well as the pull the moon has on the earth. The key to answering this is based on mass. The greater the mass removed,the greater the effect.