People are often surprised that I allow (or even encourage) people to estimate with story points as large as 20, 40, and 100. We include these values in the decks of Planning Poker cards that we sell and give away in classes and at conferences. Yet many people tell me they start out my taking the 20, 40 and 100 cards out of the deck and throwing them away. I find this unnecessary and, in some cases, detrimental to good planning. These large numbers can play a role in estimating and planning on some projects.
Let's see how. Suppose your boss wants to know the general size of a new project being considered. The boss doesn't need a perfect, very precise estimate. Something like "around a year" or "three to six months" is enough in this case. To answer this question you'll want to write a product backlog. You want to put no more effort into this than you need to. Since the boss wants a high-level estimate, you can write a high-level product backlog. Big user stories ("epics") that describe large swaths of functionality will suffice. And these epic user stories can be estimated with the large story point values. Do you want to do this on every project? No, definitely not.
There are some projects where when the boss asks for a plan you better be close. "Heads will roll if you're wrong," the boss announces on those projects. On those projects you don't want to estimate epics and, therefore, won't use the large values. Other projects (such as contract development) won't have the tolerance for error that may come when estimating epics and using values such as 20, 40 and 100. But some projects can tolerate that amount of error in the estimates. Mis-estimating a few epics is probably not enough to change the answer we give a boss who just wants to know if we think a project can be released in Q1 or Q4 next year.
Removing the large values from a deck of Planning Poker cards is like deciding to strike "millions" and "billions" from our vocabulary just because our bank balances are only in the thousands.