McGoldrick quit Duncansfield last week to return to Stenhousemuir as assistant to new Warriors boss Davie Irons, having been part of Irons’s management team during his previous spell in the Ochilview hotseat.
Kilsyth moved quickly to secure the services of Kelly who has been coaching at East of Scotland League side Dunipace.
Kilsyth secretary William Dunbar admitted the club couldn’t stand in McGoldrick’s way.
He said: “It came out of the blue but there was no way he wasn’t going to go.
“He’s worked with Davie Irons before there so as soon as Davie got the job he approached Kevin to join him.
“Gary was the first person I approached. He stood in for a few games prior to us appointing Kevin but just didn’t feel he had the experience at the time to take the job.
“But since then he’s been at Dunipace taking the under-19s and under-21s for the last 18 months so he’s got a handle on running teams and is more than happy to take the team on.”