Author Topic: June 8th Hearing  (Read 12520 times)

0 Members and 1 Guest are viewing this topic.

Offline nomatter_nevermind

  • Hero Member
  • *****
  • Posts: 5449
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #30 on: June 08, 2013, 09:40:48 AM »
O'Mara on Doddington's direct. He and West have divided up the experts, instead of one of them specializing in that area.

Offline jjr495

  • Full Member
  • ***
  • Posts: 118
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #31 on: June 08, 2013, 09:45:42 AM »
Oh No. Omara is confused and is confusing his expert. Bring back West. Owen talked about Gaussian mixture models a, a very different issue.

Offline RickyJim

  • Hero Member
  • *****
  • Posts: 1580
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #32 on: June 08, 2013, 09:53:42 AM »
I think Doddington is there to talk about standards for speaker detection and how Owen and Reich don't come close to the accepted standards.

Offline nomatter_nevermind

  • Hero Member
  • *****
  • Posts: 5449
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #33 on: June 08, 2013, 09:58:15 AM »
Wikipedia says NIST is 'a non-regulatory agency of the United States Department of Commerce'.

Quote
NIST had an operating budget for fiscal year 2007 (October 1, 2006-September 30, 2007) of about $843.3 million. NIST's 2009 budget was $992 million, but it also received $610 million as part of the American Recovery and Reinvestment Act.[2] NIST employs about 2,900 scientists, engineers, technicians, and support and administrative personnel. About 1,800 NIST associates (guest researchers and engineers from American companies and foreign nations) complement the staff. In addition, NIST partners with 1,400 manufacturing specialists and staff at nearly 350 affiliated centers around the country.

Offline RickyJim

  • Hero Member
  • *****
  • Posts: 1580
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #34 on: June 08, 2013, 10:48:01 AM »
Doddington thinks that it would be almost impossible to come up with a methodology to test how well a system could compare extreme screams with speech or “simulated screams”.  Ethically getting the data would be a problem.  Thus I think this means Owen is out via Frye.

Offline Mojo56

  • Newbie
  • *
  • Posts: 4
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #35 on: June 08, 2013, 10:51:09 AM »
I just wanted to add to the list of TY's to cbolt. Awesome job. Thanks for taking the time to do this.

MOM just asked Doddington if he heard the testimony yesterday. "Yes, unfortunately". LOL. 

Offline RickyJim

  • Hero Member
  • *****
  • Posts: 1580
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #36 on: June 08, 2013, 11:03:46 AM »
The recording(s) Reich analyzed was really 8 bits, not 16 like Reich thought was necessary. 

Offline cboldt

  • Hero Member
  • *****
  • Posts: 1262
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #37 on: June 08, 2013, 11:04:34 AM »
Doddington says that a scream being identified is ridiculous.  Speech is a performance.  He says he is just an electrical engineer, but he does not know how the voice system changes, going from normal speech to a scream.

O'Mara asks Doddington to define the community of speech experts.  He notes a couple people in Australia who would probably disagree with French, as the Aussies are more in the "hard science" mold.  Then he moves to the community that Doddington is in - he says hard science, probability and statistics, trying to quantify reliability.  O'Mara asks how these two communities differ.  Doddington said the difference is between the forensic people making decisions, the scientific community is evaluating the reliability of those decisions.  Doddington expresses frustration that the forensic people are not self-critical enough.  Doddington mentions NIST's standard in the area of Human-aided Speech Recognition (HASR), and the fact that there are millions of trials (speech comparisons); and HASR calls for 15 standard trials or experiments that software can be tested against.  The speech examples were 2 1/2 minutes of speech for each.  This was considered a difficult test, because the samples were from phone or other media.  Doddington says maybe making the speech samples shorter should have been done too, as another way to make the NIST/HASR evaluation useful to distinguish between the speaker analysis softwares.

O'Mara asks about identification of screams.  Fool's mission.

It would be a challenge to create the data to test the evaluators.  "How do you get the scream data?  You have to put people under an unethical amount of stress."

He describes the Lombard effect.  People speak louder in a noisy environment.  This was simulated with headphones, so a speaker would automatically, without knowing it speak louder.  The reverse was done too, getting people to speak softly.  He doesn't have the numbers with him, but speaking louder/softer had a significant effect on the performance of the voice recognitions systems.

Increase the distance (of speaker) from the microphone, the error rate goes up.

Asked if he heard Owen's testimony about looping.  "Yes, unfortunately I was"  "Doing that is ridiculous"  You aren't adding thing, there is no basis to say that repetition will improve the performance.  It's a violation of common sense.  Statistical dependence, which the expert now has to define.

Asked about pitch change.  Witness says he was surprised by the mildness of French's rebuke.  If you raise the pitch, you change the form of the frequencies, there is absence of basis to compare the format frequencies.  On a pitch manipulation like that, the witness would not expect a match.  O'Mara adds, which is exactly what Owen's conclusion was, no match.

Asked if he heard a recording of the NEN call.  Yes.  Says programs that display waveform and spectrograms are all the same (which is basically true, the math is simple).

He addresses "cleaning up" of signal to isolate the speech.  He says it is frustrating exercise, unless there is some discrete noise like a 60 Hz hum or a whistle, any constant tone, that can be suppressed and improve things, but even in it is not possible to improve the intelligibility of speech.  The human ear is magical in being able to process speech, although sometimes it hears things that aren't there.  Nothing Reich did would improve the intelligibility, and probably made it worse.  And that is the understanding of the scientific community.  Could you hear what Reich heard?  Laughter.

Asked about listener bias (hearing what you are preconditioned to hear), "Apparently for Dr. Reich, this was very effective."  "Just the imaginary stuff, right?"  O'Mara says we'll leave it to the court to decide that.  The expert laughs and says "this is absurd."  O'Mara takes this as a "no."  The witness says that Reich's attached importance of 16 bit data is weird - so the witness looked at the data, and it is 8 bit data.  The top 8 bits in the recordings is all zeros.  Linear PCM data.  16 bit format, 8 bit data.  LOL.  Now, I understand exactly what he means.  O'Mara doesn't get it.  Gets to a question, does this degrade the reliability of the underlying analysis?  Yes, but not the reason Reich's opinion sucks.  IOW, 8 bit sound data can be plenty good enough.

I like this witness, but I'm a pretty dry fellow, myself.  He's being very patient with O'Mara, and it is not easy for him to be patient.

Offline annoyedbeyond

  • Hero Member
  • *****
  • Posts: 1407
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #38 on: June 08, 2013, 11:10:17 AM »

I like this witness, but I'm a pretty dry fellow, myself.  He's being very patient with O'Mara, and it is not easy for him to be patient.

Should be fun to see what happens with Mantei.

Offline cboldt

  • Hero Member
  • *****
  • Posts: 1262
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #39 on: June 08, 2013, 11:13:23 AM »
Nelson sustains a state objection to O'Mara question about genesis and history of some sound analysis software.  The objection was to relevance.  O'Mara rejiggers his line of questioning in response.

Is Doddington restricted in what he can say about his work at the NSA?  Doddington has Top Secret clearance.  There are compartments within that.  Doddington advised NSA in the area of speaker recognition.  O'Mara is done with direct exam.

Nelson wants to put the feed bag on now.  Mantei thinks about an hour of cross exam and redirect.  The witness is told that he may not discuss his testimony with anybody, that includes the lawyers, he is still under oath.

Offline cboldt

  • Hero Member
  • *****
  • Posts: 1262
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #40 on: June 08, 2013, 11:20:22 AM »
Should be fun to see what happens with Mantei.

Yes.  Doddington was impatient with O'Mara, and he's apt to be less so as the absurdity of the questions is increased.

I do think it is funny that Reich's 16 bit assumption was false.  Reich says a 16 bit resolution was critical to the success of his investigation (a point that Doddington would disagree with), and Reich was working with 8 bit data.  Riech's report does a good job of describing the difference, as the range of possible loudnesses in any given "chunk" of the thousands of pieces of sound data in any second.

Offline jjr495

  • Full Member
  • ***
  • Posts: 118
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #41 on: June 08, 2013, 11:36:09 AM »
Yes.  Doddington was impatient with O'Mara, and he's apt to be less so as the absurdity of the questions is increased.

I know many smart government scientist like Doddington. They are not used to being asked questions by technically uniformed people. Professors have to deal with such questions all the time.  French's teaching and courtroom experience certainly helped his testimony.

Offline RickyJim

  • Hero Member
  • *****
  • Posts: 1580
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #42 on: June 08, 2013, 11:55:57 AM »
Even after a couple of minutes, it is clear that Mantei is much more tech savvy than O'Mara.

Offline cboldt

  • Hero Member
  • *****
  • Posts: 1262
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #43 on: June 08, 2013, 12:16:58 PM »
13:48 - Court is back in session.  I was ruminating to myself, about the confluence of junk science and junk law.  I can imagine that Doddington has very low regard for the legal system, at least this case, as the scream evidence is being entertained as amenable to expert opinion for screamer ID.

Mantei asks about hourly rate - Doddington says today is $2,000.  He charges $100/hour than Dr. French.  This is the first time you have testified as an expert?  No, once before, 30 years ago, in Minnesota.

Off the top of his head, Doddington remembers the name of the sound tool he used, "Audacity."

Mantei says, you don't do forensic analysis.  Doddington agrees.  Doddington says his function is to evaluate the performance of the various tools that the forensic voice analysts use.  Jeff Morrison in Australia is more interested in probability models than Dr. French.  So, says Mantei, it is not unusual for scientists to have disagreement.  Doddington says disagreement is good, that's where we get progress.  His role is to evaluate the disparate claims and decide which one is right.

Doddington's first reaction to the 911 sample was similar to French.  Data of this sort is not going to provide a viable source to make a decision of any kind.  Mantei says that Doddington got the result he expected, which is that the source sound file is insufficient to reach a conclusion.  Doddington reads Reich's report, and that stoked Doddington's curiosity, so he listened to the 911 recording.  He thinks Reich's conclusions are absurd.

Doddington says that the forensic guys don't like to submit to NIST microscope, because they don't want to be exposed (as unreliable).

Mantei painting Doddington as inexperienced because he doesn't do forensic evaluations, and he hasn't done a forensic evaluation in this case either.  Doddington says his listening confirmed his impression going in.  Mantei says something about the conversation with O'Mara as to listener bias.

Mantei asks what is the best system you have come across?  Doddington says there is a slow evolution of algorithmic approaches ... GMM / Gaussian Mixture Modeling, the rage today is I-Vectors, but all of the systems are similar, pretty damned good.  Both GMM and I-Vector are probability models.  Mantei asks if any of the analytical tools will be 100% accurate.  No.

Mantei moves on to standards.  Doddington tries to stay away from them.  NIST gives the people the data it will work with.  NIST is not in the business of setting standards for speech examination.  Mantei gets Doddington to agree that minimum words or minimum time is not an ultimate gauge of suitability of the sample.

All Mantei has is sophistry.  Nelson will lap that up.

O'Mara should ask the extension of Mantei's closing question.  What result would you expect a forensic examiner to get, what probability of success, using the 911 call sample?  That is not the same as asking what the result will be, it is asking about the reliability of reaching a conclusion.

O'Mara's current focus on the function of NIST has no benefit in the goal of disqualifying Owen or Reich.  I can't figure out what his point is, you can just about guarantee that Nelson is lost.

Offline RickyJim

  • Hero Member
  • *****
  • Posts: 1580
  • Rate Post +0/-0
Re: June 8th Hearing
« Reply #44 on: June 08, 2013, 12:21:22 PM »
NIST is about determining whether or not a methodology is actually reliable, not fixing the methodology.  O'Mara is not doing a good job showing that Owen's and Reich's methods have never been evaluated by NIST.  Also they are so off the wall that they wouldn't be considered as a subject for evaluation.  I think behind this confusing stuff is what exactly the Frye test is.

 

Site Meter
click
tracking