Critical Analysis - Hints for Critical Analysis in Class Activities and Daily Life
by Jerry Mertens, Dan Scriver, and Dan Knip, St. Cloud State University July 2006
This article serves as a starting point for developing a specific repertoire for critical
analysis when one encounters extraordinary claims. The object of this material is to help
develop a critical analysis repertoire that can be used to analyze the critical analysis
events we typically use at the start of class, but more important, to develop critical
analysis skills for use in daily life. False explanations, inadequate analysis of human
behavior, extraordinary claims, fraudulent activities, etc., all must be carefully analyzed in
order to find the variables at work in a given situation. The frequent use of deceptive
practices exists. The number one source of white collar crime in 1994 was financial
scams according to Channel 9 News in Minneapolis, as reported on May 5, 1994.
Students in the class witness magic tricks that are presented as psychic events. The
student's job is to become a detective and offer real world explanations for these
supposed psychic events. This handout is intended to help students develop and utilize
some critical analysis skills.
1. Utilizing a Useful Definition of Magic in Critical Thinking:
When one really analyzes magic, it is the use of scientific principles which makes it look
like scientific principles are not at work. This definition of magic, developed by the first
author, is useful in the critical analysis of events performed at the start of class, as well as
when critical analysis is needed in daily life. Hobby and stage magicians do exactly this.
They use scientific principles to make it appear as though there is NO use of scientific
principles in the performance of the trick. Magicians and psychics try to perform their act
so it does not look like an application of science. When the stage magician says that
they will pull a rabbit out of an empty hat, the magician does NOT pull a rabbit from out of
nowhere. The magician uses principles which make it look like they are pulling the rabbit
out of an apparently empty hat. In another trick, the magician is not really sawing
someone in half. When they use a good set of principles, it just looks like the event is
done by "magic."
A con-artist does this also. The following is often heard after someone has been made a fool of
by a con artist, "Jo(e) Blow was such a nice person. I just can't imagine that this person
was involved in such a scam." Of course, if the person did appear that way, (like a con)
you would not buy into the scam. You would not expect the person to say, "Jo(e) Blow
looked like a crook from the moment I first saw him/her. I knew that it was a scam, but I still gave
them all my money. I just like to give money to people who are trying ineffectively to perpetuate a
scam."
Approaching the unknown, examining extraordinary claims, questioning inferential
explanations, etc., is needed. This definition of magic encourages the search for the
scientific principles being used. What other principle(s) of science could account for what
appears to be a stunt that defies the laws of science?
2. Magic is the Exception in Daily Life Where Society Approves of ANNOUNCED Deceit,
Fraud, Lies, Cheats, etc.
The role magic plays in our society is another concept that may be useful in developing
and using critical thinking. Magic, as entertainment is society's approved way of using
fraud, cheating, etc., to fool people. The problem is that all who use magic technology
are not announcing it. Do understand that there does exist a large commercial multimillion
dollar industry out there that is working on; inventing new things, manufacturing
items to deceive, marketing devices with good sales pitches, selling sophisticated
technology, organizing groups, providing a lot of literature (propaganda) with well orchestrated
sources of information, and so on. This is not simply passing on the secrets of the art form of,"magic." It can be personal and secret information. People have tied some of the deceptive
methods of the art form of "magic" to strong ideological cult-like efforts and have organized
groups to perpetuate this cause. In short, there are many people abusing the art form of "magic"
in order to gain money and power. This may be verbal deception or magic gimmicks. If you are
going to fall for some extraordinary claim, do it with the knowledge of the technology and methods
available. Better yet, don't fall for it at all.
As far as the in-class activity, the nature of the "fraud" producing industry means that
there will continue to be a constant new supply of stunts for the student to guess at what
principles are at work. Therefore, these will be stunts that students have a minimum of
experience with. This condition maximizes the students need to compare the environment against
their knowledge to increase the chances of coming up with an adequate explanation.
3. Source of the Claim:
One needs to question the source from which a claim has come. What do you know
about the source? Is it reputable? Be careful. Just because a name sounds good or
scientific does not prove that good techniques are being used. The less familiar you are
with the source, the more reservations you should have about the validity of the claim.
As in the class procedure, you should know it is a trick. In real life, you are not warned
before a fraud, con, scam, hoax, skullduggery, swindle, or flimflam takes place. What is the
nature of the source of evidence?
4. Determine the Source of Experimental Scientific Data:
Are the experimental findings that are in question reported in reputable journals? What
values or personal opinions are evident in the source of the findings? Does it appear in a
journal published to promote the specific self interest or the position of a group? Does
the finding appear in a "refereed journal" (i.e., a journal with a panel of reputable
authorities in the field to evaluate the studies that are submitted for publication?) Is this
panel as unbiased as possible? Is it a good scientific source? Science is a buzz word, so be
careful. Some of the worst incorrect inferential fictions stated as supposed causes of
behavior have, and will be, offered as good science. You must be able to recognize false
claims.
5. The use of Independent and Dependent Variables in Experimentation:
Were the findings based on independent variable experimentation? How were the
relevant variables controlled? If it was an experimental design, was it a single-subject
design or a group design the study used? Learn the role, value, and shortcomings of
both of these research designs.
6. Has the Claim Been Replicated by Independent Sources?:
Has the finding been replicated by a number of independent sources using good
experimental methods? For example, the concept of positive reinforcement has been
replicated numerous times by independent, credible sources, making it a useful concept.
If fraud had been found in these labs, e.g., for research on reinforcement, if a string had
been used to pull the animal through the procedure or false data (the so called "dry run")
was found to be used, we should question the reinforcement notion as much as we
question extra-ordinary claims.
7. Forget Those Supposed "Gems" You Have Learned:
At some time during their education, many students are taught things like, ''Everyone
selects the color blue'' (or yellow or green etc. or, ''People are attracted to bright colors,
and will always select the brightest one." ''A certain position is always selected." "People
will go to the right all the time." "The hand is quicker than the eye." These sorts of "gems”
have no use in predicting what a person will do in a given situation.
8. What Someone Says vs. What The Person Does:
Listen carefully to what a person says he or she is doing, then analyze what the person is
actually doing. Are the two consistent? Listen and look for any inconsistencies. In the case of“fear” for example, which of the following statements are relevant:
a. Does the person say he/she is studying fear, but actually gives the subjects a paper
and pencil test?
b. Does the person use a physiological measurement (heart rate, respiratory rate, a GSR
- Galvanic Skin Resistance, etc.) and call it "fear?"
c. Does the person use a sample of fear related verbal statements, and call that "fear?"
d. Did the person just conjecture (their judgment call) ones "fear?"
e. If the person said, “you have a random choice,” do you?
These are all different examples. Know what a person's statement is based on.
9. Being Lucky vs. Using Coincidence:
One may take advantage of luck when it happens. If the real odds of an event happening
are very bad, the psychic probably is not using luck in a given prediction. Yet, many
individuals can take advantage of others when a low probability event happens.
They can make the event appear much bigger or stronger than it actually is. There is a
difference between these two. A psychic probably is not using 1 in 52 probability to
predict your selected card. Those odds work for "post" diction, not prediction! However,
if luck goes his or her way, a good magician or psychic will take advantage of it and can
make a low probability event look like a bigger miracle.
10. Is The Situation Really Controlled?:
Supposed psychics claim the situation is controlled to eliminate the possibility of any foul
play, when in reality, the relevant variable is not sufficiently controlled. A self-professed
psychic could set up a situation that appears controlled. For example, if he or she
claimed to be able to rotate a dollar bill on a stick pin using only the power of his or her
mind, the "supposed" controlled setting he or she could create might be to enclose the
dollar bill on a pin under an upside-down fish tank. It would seem that the tank would
eliminate the possibility of foul play involving moving air (like the supposed psychic's own
breath), but in a case exposed by Koran, it actually did not prevent air from going under
the tank and above the table where the tank had been placed. How could she/he
manipulate the dollar bill in this situation? Is the tank imperfect in some subtle way? Will
it rest flush (air tight) against the table when it is upside-down? Could there be a gap
between the upside-down tank and the table it sets on which could allow air to flow
under the tank to reach the dollar bill? Be observant during situations that are supposed
to be controlled. You need to question the controls in every situation.
11. Be on Guard For Misdirection:
The misdirection can be verbal or physical, and is intended to distract your attention.
Here are some examples to look out for:
a) Repeated pairings with words or actions, e.g., continuously repeating an action,
motion, etc. This will cause your attention to be dulled so you can be more easily fooled.
b) Pointing, and any other body movements, which gets you to look off in the wrong direction. Ask yourself why certain moves are used.
c) Cards, book pages, or virtually any object can be secretly manipulated by the
supposed psychic. If the first sheet of paper is shown as blank, that doesn't mean a
second sheet of paper introduced is also blank, or that he/she didn't write, not noted by
an absence, something on it after revealing it was blank.
d) One may be misdirected as readily by verbal comments as by movements, gestures,
looking in a certain direction, eye contact, etc.
e) Gaps in a verbal account are left for you to fill in for yourself, leading you astray. For
example, a person opens up a "new" box of playing cards and pulls out a stack of
apparently 52 regular playing cards. Before you get led astray, you might ask yourself a
question, "Is this supposed fresh sealed deck a misdirection used to get one off track?"
12. Is the Explanation You Came Up With Feasible?:
a) Is today's technology in the specific area in question as advanced as your supposed
explanation? After one of two coins appears to pass through a solid rubber mat, stating
that, "The coin breaks through the mat, then the mat instantaneously heals/seals itself
without a blemish, and this all happens before a penny that was sitting on top of the
quarter can make it through the mat?" This explanation is probably not feasible.
b) Could the psychic have placed 30 different viable predictions all over the building so
as to cover all prediction possibilities, and make it still appear as psychic ability? For
example, one in the waste basket, one upstairs, one 2 offices down the hall, one under
the table, one inside of a book in the hall, one downstairs, etc. Would this be an effective
way of performing a stunt with a supposed prediction? (Probably not. That would appear
too phony.) Is there a more feasible explanation, or is there a way for you to refine or
readjust your guess so as to make it usable?
c) Did the psychic touch the item, or was it handled by audience members? Could there
be confederates in the audience?
d) Assume there is an ordinary explanation to an extraordinary event. Often, you will not
be aware of the scientific principle(s) used. This only means that the supposed psychic
or magician has done a good job at keeping the truth hidden.
13. Realize Who is in Control:
The routine was created by the psychic performer, so he or she is the one in control of
what happens. What are they still able to control? Has she or he told you ahead of time
what is going to happen, or what she or he will do? If so, look for small changes the
person might make in the routine as it goes along. Changes may occur depending on
what you do.
14. Task Analysis:
By task analysis, one breaks down what was seen or what is happening into component
parts. To do this, you must pay close attention to everything that happens during the
episode. To miss one small, but crucial part of the stunt could lead you in the wrong
direction. In your task analysis, start with the first thing and work forward through the
whole routine. If the purpose of a task seems to be to distract you, you might ask
yourself what it is that you are not supposed to notice? If the presentation remains a
mystery after analyzing each task or component part, reconsider if you may still be
missing part of what happened.
15. Question Your Own Observations:
a) Question what is said instead of being absorbed into it.
b) Examine all the facts presented and be selective of which facts you believe to be
true.
c) Consider all the alternatives to your tentative solution.
d) Look for contradictions in the facts being presented to you.
e) Be open to change your explanation for the observed mystery. Be careful not to
become so set on an explanation that you continue to hold on to it even in light of
alternate evidence.
f) Evaluate your own biases and work toward being more objective.
g) Realize if your conclusions are based on the correlation methods. Events that are
related or correlated are not necessarily causative. Look for an independent variable at
work in the situation.
h) Push yourself to consider, “what physically possible ways exist for each of the
component parts to change in order to accomplish the questioned happening?”. What in
the physical world has to take place for the observed event to be an outcome?
i) If all facts are not available, wait for them before making a final assessment. It is better to be
empty (in need of answers) than to be full of garbage (wrong answers)! One needs to accept
questions and not faulty answers.
16. Multiple Causation:
A reaction or result can be obtained by more than one means. You may know one way a
fraud, mystery, trick, etc., is perpetuated, but that does not mean it is the only way. In
other words, there is more than one way to bend metal, levitate objects, know a selected
object, etc.
17. Be Alert:
Did you tune into the current problem late? Did you start looking soon enough? Are you currently
seeing all of what is taking place? Have there been some events which have taken place prior to
what you are now watching, e.g.:
a. this could have been the same two people meeting.
b. a confederate and the person now involved having minimal contact.
c. information obtained from friends in a variety of situations.
d. information received by the supposed psychic, but not known to you. For example, overhearing a conversation in the lobby, street, entrance, rest room, public records, etc. Or
perhaps all in the audience were asked to write something, but some were asked to write
something different from others, etc. It has been stated that what you don't know is not a problem.
What is a problem is when what you know is incorrect knowledge.
Scientists can be vulnerable to paranormal claims and extraordinary events if they are not
careful. Often, the scientist waits too long to begin questioning the situation. This questioning
must occur continually. The task analysis needs to start with the very first task which may occur a
long time before the stunt is even performed.
The next time you receive a notice in the mail that you have won a prize for some contest
you never entered, and that all you need to do is order some product or send in a "small"
amount of money to receive your "big" prize, remember that the company or organization
may only be concerned with making a profit. The prize you supposedly won may be
worthless, and receiving it is probably contingent on you doing something that will benefit
the agent.
18. Life Can be A Stage:
Even if you may not be theatrically inclined in the way you present the world, realize
others may be very theatrical in the way they tell you about the world. There is
documented scientific literature that good theatrical suggestions have led a non attention
seeker to confess to a murder that the person had not committed. Often times, when
people describe a previous event, they tend to exaggerate certain aspects. It is more
enjoyable to tell a shocking or interesting story even if it requires stretching the truth.