I (27F) recently flew with my husband (28M) and our 2 kids (2F and 6moM). We chose to risk it in terms of seat selection (between all of us, it would have been an extra $100+ to get the option where we could pick seats).
The airline assigned us all seats together on the departing flight, but on the return, the flight was full and we were assigned a middle and window seat and then the middle seat in the row ahead. We chose for my husband to take the solo middle seat and for my toddler, the baby and I to sit in the adjoining seats so that I had a little more room to nurse the baby.
On the flight, I ended up putting my toddler in the middle seat behind my husband which was next to a man who probably was in his mid 50s or so. He had his laptop out for the majority of the flight and seemed like he was trying to work.
During the 4 hour flight, my toddler was fairly well behaved, but she was very interested in his screen and I had to redirect her a few times not to touch it. But for the most part, she stayed in her seat and played with the toys and books I had brought for her. We had to get up to use the bathroom 3 times, so we had to squeeze past him, and every time, he seemed a little more irritated at us bc he had to get up and let us exit the row and then go back. After the third bathroom trip, he asked the flight attendant if there was another seat he could be moved to, but the flight attendant said the flight was full.
My husband overheard the exchange and was mortified. He (my husband) told me I should switch seats with our toddler and I said no, the flight was almost over. I got dirty looks from the guy in our row for the rest of the flight and after we deplaned, my husband said I should have just switched seats with our toddler even though the guy was an asshole. I told him if the guy was so against sitting next to a kid, he could have switched with my husband.