Osborn began his career working in local clubs, then played on a hit record by singer Dale Hawkins. He moved to Las Vegas at age 20, and spent a year playing backup for country singer Bob Luman. With legendary guitar player Roy Buchanan among his bandmates, Osborn switched from guitar to electric bass. In 1960, with Allen "Puddler" Harris, a native of Franklin Parish, also in northeastern Louisiana, and James Burton, originally from Webster Parish, he joined pop star Ricky Nelson's backup band, where he spent four years. His playing on such Nelson hits as "Travellin' Man" began attracting wider notice, and he found opportunities to branch out into studio work with artists such as Johnny Rivers.
When the Nelson band dissolved in 1964, Osborn turned to studio work in Los Angeles full-time. For the next ten years, he was considered a "first-call" bassist among Los Angeles studio musicians (known as The Wrecking Crew), and he worked with well-known producers such as Lou Adler and Bones Howe, frequently in combination with drummer Hal Blaine and keyboardist Larry Knechtel—the combination of Blaine, Osborn and Knechtel have been referred to as the Hollywood Golden Trio. His bass can be heard on many of the hit records cut in Los Angeles during that time, along with numerous film scores and television commercials.
His playing can be heard on records by such well-known groups as The Mamas & the Papas, The Association, The Grass Roots and The 5th Dimension. Osborn can be heard on Simon & Garfunkel's "Bridge over Troubled Water" and the 5th Dimension's version of "Aquarius/Let the Sunshine In". A song featuring prominently mixed bass in melodic counterpoint to acoustic guitars is the 1972 hit single "Ventura Highway" by the group America.
Osborn played on many of Neil Diamond's major hits in the late 1960s and early to middle 1970s, including the hauntingly unique bass lines on "Holly Holy" in 1969. Osborn is also known for his discovery and encouragement of the popular brother-and-sister duo, The Carpenters.
In 1974, Osborn left Los Angeles and moved to the country and western capital, Nashville. He continued an active studio career, playing behind such vocalists as Kenny Rogers, Mel Tillis, and Hank Williams, Jr. One count listed Osborn as bassist on fifty-three number one hits on the country charts.
Osborn's instrument throughout most of his recording career was a 1960 Fender stack-knob Fender Jazz Bass, which was given to him by Fender just prior to touring in Australia with Nelson. Osborn said he was initially disappointed that Fender had not sent him a Fender Precision Bass, which he had been using, but he said he grew to like the Jazz Bass because the narrower neck made it easier for his short fingers. He strung the bass with LaBella flatwound bass strings that he did not change for 20 years and his style is distinctive, with a resonant, bright tone produced, in part, by his use of a plectrum (pick).
Many producers and arrangers chose to spotlight his contributions by mixing the bass line more prominently than had been customary, and incorporating brief bass solos into their arrangements.
He had a signature bass, named the "Joe Osborn Signature", made by American guitar manufacturer Lakland; however, it is now called the "44-60 Vintage J Bass". In 2012, Fender Guitar built a custom Fender Jazz Bass for Osborn according to his desired specifications. He recorded with this bass for the first time when producing and playing bass on teen musician Matthew Davidson's debut recording.