Dating man that is separated?

Husband died 3 years ago and been dating but The man I really want to be with is separated. I waited a while to be with him after husband died. He knew my husband before he died I've known them for about 9 years. He is not with his with she lives in a different house. She knows we see each other. They have 2 children together and they do need to spend time together to talk about the kids. I've even sat for them when they have gone out together. There is no chance they are getting back together, The only reason they are only separated is finances his kids are Autistic and he wants to take care of them. He spends a lot of time with his son. And his daughter as well which means he has to spend time with his wife. and not as much time with me. I understand his situation he can only spend so much time with me. We see each other couple times during the week. depending one school and activities. during summer. Most weekends he is doing something with his son. I am lonely sometimes and he tries to make it up to me by about once a month he spends couple days with me. Kid free. But I feel guilty because he does this usually after I say I wish he could spend more time with me. I was married to the same man for 25 years so dating has changed for me. And this person has been a friend for 9 years. When husband died he just never left me. I am free and I can date whomever I want but I really just want to be with him. Am I crazy to stay in this situation or should I move on?