You are right if the neutron star has 3 to 7 solar masses. In this case a relative cold and nonrotating neutron star will demage its structure. It is a question.

What about a star with 7 and more solar masses ? When it become Black Hole like Object (just before the creation of the Event Horizon) all neutrons oscillate with a maximum velocity close to speed of light because they are accelerated by a strong gravitational field. This oscillation causes collisions with a relativistic velocity - it causes creation of the particles/antiparticles and annihilation. A gamma radiation can't go out because it is a Black Hole Like Object, There is a redshifted radio emision only and this object absorbs much more energy from an accretion disc and whole environment.

The neutrons are oscillating and can not stop because they even have more kinetic energy because more energy is absorbed than emitted.

The neutron oscillate and rotate like in gasoeous nebula. This cloud may collapse into a neutron star (there is enough mass) but it has to emit its kinetic energy. There is no other way. It is clear I think.

A big mass is not a sufficient cause.
The main cause is to stop oscillation and rotation - extract the kinetic energy.
If the kinetic energy is not extracted (emitted) like in BlackHole Like Object the collapse stops..
.