What is the best practice when unit testing methods that have assertions instead of exceptions?

Let's say you have a public method like this

public Sprite spriteAt(int x, int y) 
{
    assert withinBorders(x, y) : "PRE: x and y should be inside the borders";

    return tileAt(x, y).topSprite();
}

Versus

public Sprite spriteAt(int x, int y)
{
    if (!withinBorders(x, y)) throw new InvalidArgumentException();

    return tileAt(x, y).topSprite();
}

In the bottom case, I would usually have a unit test cases that checks if an exception is thrown when invalid value for x and/or y are given like:

@Test(expected = InvalidArgumentException.class)
public void SpriteAt_InvalidX_ThrowsInvalidArgumentException()
{
    sut.spriteAt(-100, 0);
}

This test case is to ensure that the argument validating logic is implemented in the method.

However, for the assertion method at the top, I am not sure what I am supposed to do. The assertions are not production code and I think this means I don't have to test the assertions.

On the other hand, I think unit tests should notify the developer by failing when there is a change of logic in a method. If I do not write a test case for checking if there is an assertion that checks invalid arguments (just like how I do for exception method), then I may not realize I have made a mistake when I accidentally remove the assertion line of code.

Therefore, what I want to do is to check whether assertion is in place if junit is ran with assertion enabled and don't do anything when assertion is not enabled. Below code will contain pseudocode.

@Test
public void SpriteAt_InvalidX_AssertionThrowsException()
{
    if (assertion is enabled)
    {
        try
        {
             sut.spriteAt(-100, 0);
             fail();    
        }
        catch (AssertionFailureException e)
        {
        }
    }
}

So back to my point. I want to know whether unit tests are supposed to test assertions. If so, am I going in the right direction? If not, how do you prevent from accidentally removing assertion code without unit tests?