12 replies to this topic

[Sinipull]

This tutorial is about java Robot class. (java.awt.Robot)

I will quote Sun here:

Quote

"This class is used to generate native system input events for the purposes of test automation, self-running demos, and other applications where control of the mouse and keyboard is needed. The

primary purpose of Robot is to facilitate automated testing of Java platform implementations."

So, basically this gives the low-level input control over your mouse and keyboard to your java program. Note, that you are still able to control your mouse and keyboard when the robot is running (in case of correct coding).

IMPORTANT: The Robot class gives your program as much power, as you have over your computer. It can do EVERYTHING you can do with your mouse and keyboard. The program has even ability to write it's own code, while running and these kind of programs should be written very carefully. (Not because Skynet is probable to evolve, but it can wreck something on your computer).

Ok, let's see how to start using the Robot class.
First we need to import the Robot class:

import java.awt.Robot;

Then we will need to make a Robot object:

Robot r = new Robot();

Now we have access to our Robot methods. (The complete list of methods and javadoc can be found here)

MOUSE:

r.mouseMove(x, y);

which moves your cursor instantly to the specified location.
Note, that x = 0 is located in the left of your screen and y = 0 is located at the top of your screen.
The largest values, where your cursor can move, is your screen resolution - 1. On my 1680x1050, it is x = 1679 and y = 1049.
If you tell the cursor to move out of the screen area, it will just slam against the respective wall.

r.mousePress(buttons);

r.mouseRelease(buttons);

Presses and instantly releases your specified button.
IMPORTANT: Never forget to release the button after pressing. (Can mess up something on your computer as a result, if lot of that is done)

the mousePress() and mouseRelease() takes int as an argument, which should be selected from amongst the constants in the InputEvent(java.awt.event.InputEvent) class:

those are the constants:

InputEvent.BUTTON1_MASK  //left button

InputEvent.BUTTON2_MASK  //middle button

InputEvent.BUTTON3_MASK  //right button

(I'm not sure, how they depend on the mouse, platform, configuration, etc, so you should check them out yourself.)

r.mouseWheel(wheelAmt);

This scrolls your mousewheel, if you have one. The wheelAmt argument is an int and should be positive or negative int value, depending how much or which way do you want to scroll the mouse.

KEYBOARD:

r.keyPress(key);

r.keyRelease(key);

Presses the specified key and releases it after.
IMPORTANT: Never forget to release the key after pressing. I actually had to restart my computer to fix the damage, that was caused by this mistake. Restarting just Eclipse didn't help.

The "key" argument is again int and should be selected as a constant from KeyEvent(java.awt.event.KeyEvent) class. Now, there are too many constants to copy them all here, so i'll just show you how they look:

r.keyPress(KeyEvent.VK_A); // press A-key

r.keyRelease(KeyEvent.VK_A); // release A-key

If you use proper IDE, you are able to find them all easily.


OTHER:

Robot class can also make screenshots or get the pixel color on screen:

r.createScreenCapture(new Rectangle(0, 0, 100, 100));

Creating screenshot requires Rectangle (java.awt.Rectangle) object as an argument, as this is the specified area, from where the screenshot is taken.
The result is BufferedImage (java.awt.image.BufferedImage) which can then be handeled properly (analyzed, saved or whatever needs to be done).

r.getPixelColor(x, y);

Gets you the pixel color as Color(java.awt.Color) object.

r.delay(ms);

Sleeps the specified amount of milliseconds. Is very useful when writing a Robot program. Is pretty much interchanceable with Thread.sleep(ms).

USEFUL STUFF TO KNOW WHEN WRITING A ROBOT

Get current screen resolution:

Toolkit.getDefaultToolkit().getScreenSize();

returns a Dimenson(java.awt.Dimension)

Get current mouse position

MouseInfo.getPointerInfo().getLocation();

returns a Point(java.awt.Point)


---

Again, beware of writing programs that take over your mouse and keyboard without sleeping. The only possible way to stop it, is the restart of the computer, from the physical button.
It's easy to stop the program from task manager, if only your mouse is stuck somewhere, but if you keyboard is being used too, then you are pretty much forced to plug off your computer.
Therefore, loops are dangerous, if not used properly!




The Robot class may not work on some platforms, as Sun has written:

Quote

Note that some platforms require special privileges or extensions to access low-level input control. If the current platform configuration does not allow input control, an AWTException will be thrown when trying to construct Robot objects.

Have fun ;)

[Cander]

Good tutorial! rep+. You could have linked this page though, the full documentation of the java.awt.Robot class which tell all existing methods etc.

http://java.sun.com/.../awt/Robot.html

[Sinipull]

Thanks. :)
But i did link to that page, check right above the Mouse chapter.

[Cander]

Oh, sorry! missed that

[Prog4rammer]

Good and useful tutorial rep+ Thanks You :) ...

[unscripted_1]

so would you use r.getPixelColor() to try to get Robot to react to things happening on the screen?

Like could you create a boolean function, such as hasSomethingHappened() which keeps checking the pixel color, & just get the program to keep sleeping till it returns true?

Is there any more efficient way to get Robot to interact with screen events?

[Sinipull]

I wouldn't use r.getPixelColor(). I would use r.createScreenCapture() and get the Colors from there, i'm pretty sure, it is more efficient way for large data access than getPixelColor().

You have to create your own artificial intelligence for the machine, no 'easy way' here. It's very hard to create something that could actually understand what's on the screen. Neural Networks would be one possibility.

[wim DC]

Quote

Again, beware of writing programs that take over your mouse and keyboard without sleeping. The only possible way to stop it, is the restart of the computer, from the physical button.
It's easy to stop the program from task manager, if only your mouse is stuck somewhere, but if you keyboard is being used too, then you are pretty much forced to plug off your computer.
Therefore, loops are dangerous, if not used properly!

This has happened to me several times :( i let the robot control the mouse and click with it. But i got stuck in a loop several times and because it was constantly clicking i couldn't kill the process with the keyboard because task manager would lose focus. So now i always set up a timer which stops the program after a set time (mostly about 10-20 seconds) ;)

[GMVResources]

Nice job Sinipull!

[linda]

@Sinipull, I am trying to build up an automation program to do some specific jobs.
Considering the capabilities of robot class, it works fine but i am stuck at two things:

1. Is there anyway to know if a process is finished in another application before pressing a key for example.
Like, if i run e.g a windows media player, how would i know it is loaded in OS and ready to be used, before
i click further using robot.xxx

2. Using robot, we can read the mouse and write back both mouse/keyboard actions. But is there anyway to
read the keyboard as well?
Like, if i automate documets and want my robot class to wait for some specific input from user, for (say) a
particular field, and rest it has to fill automatically, how can it read the key input?

Can someone please answer this?

[Cander]

linda said:

@Sinipull, I am trying to build up an automation program to do some specific jobs.
Considering the capabilities of robot class, it works fine but i am stuck at two things:

1. Is there anyway to know if a process is finished in another application before pressing a key for example.
Like, if i run e.g a windows media player, how would i know it is loaded in OS and ready to be used, before
i click further using robot.xxx

2. Using robot, we can read the mouse and write back both mouse/keyboard actions. But is there anyway to
read the keyboard as well?
Like, if i automate documets and want my robot class to wait for some specific input from user, for (say) a
particular field, and rest it has to fill automatically, how can it read the key input?

Can someone please answer this?

1. Nope. You can use Thread.sleep() while a started program by robot is starting up to wait for it to load, but there is no is it guarantee the started program is ready to use after sleeped time. Like if your computer has a slow day.

[Xdawn90]

This is something new for me. Nice tutorial.