Ultrafast all optical magnetization switching in GdFeCo layers on the basis of Inverse Faraday Effect (IFE) was demonstrated recently and suggested as a possible path toward next generation magnetic data storage medium with much faster writing time. However, to date, the demonstrations of ultrafast all-optical magnetization switching were performed with powerful femtosecond lasers, hardly useful for practical applications in data storage and data processing. Here we show that utilization of IFE enhancement in plasmonic nanostructures enables fast all-optical magnetization switching with smaller/cheaper laser sources with longer pulse durations. Our modeling results predict significant enhancement of IFE around all major types of plasmonic nanostructures for a circularly polarized incident light. Unlike the IFE in uniform bulk materials, nonzero value of IFE is predicted in plasmonic nanostructures even with a linearly polarized excitation. Experimentally, all-optical magnetization switching at 20 times lower laser fluence and roughly 100 times lower value of laser fluence/pulse duration ratio is demonstrated in plasmonic samples to verify the model predictions. The path to achieve higher levels of enhancement experimentally is discussed.