Micro-patterned hydrogels have received increasing attention in various research fields that need a fine structure and advanced functions compared to bulk hydrogels. For enhancing their performance, control of the size and distribution of pores in hydrogels is crucial. In particular, for the application of thermoresponsive hydrogels to a soft actuator, the characteristics of pores play an important role in enhancing the response time. We formed a porous structure in micro-patterns of polydiethylacrylamide (PDEAM), a thermoresponsive hydrogel, and analyzed the effect of pore size on the response time of the patterned hydrogels. Micro-patterned thermoresponsive hydrogels are fabricated by photo-crosslinking PDEAM copolymerized with benzophenone photo-crosslinker and polystyrene (PS) as a porogen. Pores sufficiently smaller than patterned objects, between a few micrometers to a few tens of micrometers, are successfully formed by controlling the content and molecular weight of PS. As the size and number of pores increase, the response time is improved, and the response time for swelling and deswelling is improved by up to 52 and 43% by blending PDEAM with 50 vol% of low molecular weight PS (5 kg/mol). This simple way to form a sub-millimeter scale hydrogel structure with controlled pores can be utilized in the emerging research fields, including 3D cell scaffolds, targeted drug delivery, and soft robotics.