I work with many hazardous products. Solder is no different to anything that is bad for you. If you are thinking it is only minor and won't kill you, you may be right. However, it could greatly affect the quality of your life. Just using a fan is no good unless you have air exchange out a window. Without a window it will diffuse into the air and reduce the levels, but increase the area to include your wife and kids. It is a cumulative group of toxins that may include lead or not. These toxins remain in your body for a long time and every time you breath it, you add more that stay. I use a chemical mask on heavy days or cold winter months. This is the only way to protect yourself unless you use a air make-up/exaust system. Air make-up/exaust systems protect everyone in the house as well, not just you.
A simple project that should cost less than $50 and if you do a lot of soldering or even hobby painting, may save your quality of life. The idea is to buid a 30"wx24"hx24"d(approx.), wood or acrylic table top exaust box like a spray booth. You put an exaust fan ported out a window on the back side and 6" from the back use a furnace filter to diffuse the suction. You can mount lights on the inside and hooks on the outside for tools. How much airflow is up to you, variable is great as well. Close the door in the room you are working to create a negative pressure zone so that the fumes want to go out the exaust and not out the door. Air from under the door will create the needed air make up in the low pressure area to stop the fumes from leaving the room to contaminate the rest of the house. Seal around the exaust at the window opening as best as possible to stop the fumes from coming back in, keep you warm in the winter, and stop the pressure of the room from fluctuating too much. When was the last time $50 made fun things safe?