"Allergenic Molds" causing allergic sensitization, which is an immune response from the human body.  Those responses include  symptoms such as runny, itchy nose, congestion, sneezing, watery eyes and chronic fatigue, insomnia etc, in all of which immune reaction --inflammation or pre-inflmmation reactions happens.
All molds, dead or alive, even the dead broken fragements, tend to cause allergenic reactions from the human immune system if their spores, conidia or propagules, dead fragments, are inhaled. The immune system recognizes a specific invader by tagging antigens on them when they first appear inside a human body, then destroys them as the first defense and as a memory for future encounters.
"Pathogenic molds" are those causing "Subcutaneous Mycoses" and "Deep organ mycoses" in which the human immune reaction wrestles against the invading molds in the former (Subcutaneous Mycoses), while gradually lost in the latter form (Deep organ mycoses). Any mold that is capable of growing in 37 C (i.e. the human body temperature) are potentially a pathogenic mold!!!
Those molds causing "Subcutaneous Mycoses" are relatively weak pathogens, that can only infect shallow human skin, or strong pathogens having difficuty of  infecting a relatively immuno-competent persons. Those molds are like: Acremonium, Pseudallescheria (Scopulariopsis in anamorph),  and black yeast Exophiala etc.
Those molds causing "Deep organ mycoses",  are the real dangers. All those BSL2-BSL3 molds are capable of growing at 37 C! Some are dimorphic, namely, are capable of growing as filamentous molds or as yeasts.

The culprit list --most dangerous molds -- include:
Paracoccidioides brasiliensi;
Blastomyces dermatitidis;
Coccidioides immit­is
(Valley fever agent);
Histoplasma capsulatum;
Penicillium marneffei (the only Penicllium species rated as BSL-3 mold);
Aspergillus flavus (producer of mycotoxin aflatoxins); 
Aspergillus fumigatus (the No.1 pathogenic mold in clinical cases);
opportunistic yeast Cryptococcus neoformans;
another opportunistic yeast Candida albicans (the widespread yeast pathogen);
and other Zygomycetous molds Rhizopus and Mucor species (black bread molds).